%%This is a very basic article template.
%%There is just one section and two subsections.
\documentclass[11pt,oneside,a4paper]{article}
\usepackage{float}
\usepackage{graphicx} 
\usepackage{cite}
\usepackage{pdfpages}
\usepackage{listings}
\usepackage{color}
%\usepackage[titletoc.title]{appendix}
%\usepachage[dotinlabels]{titletoc}

\definecolor{dkgreen}{rgb}{0,0.6,0}
\definecolor{gray}{rgb}{0.5,0.5,0.5}
\definecolor{mauve}{rgb}{0.58,0,0.82}

\lstset{frame=tb,
  language=Java,
  aboveskip=3mm,
  belowskip=3mm,
  showstringspaces=false,
  columns=flexible,
  basicstyle={\small\ttfamily},
  numbers=none,
  numberstyle=\tiny\color{gray},
  keywordstyle=\color{blue},
  commentstyle=\color{dkgreen},
  stringstyle=\color{mauve},
  breaklines=true,
  breakatwhitespace=true
  tabsize=3
}

\title{Emotion Recognition for Mobile Devices with a potential use in Serious Games for Autism Spectrum Disorder}
\author{Adiljan Abdurihim and Olav Brenna Hansen}

\begin{document}
\maketitle

\section{Introduction}\label{s:intro}
Humans get a lot of emotional information through visual communication. This is essential for our understanding and interpretation of emotions. The understanding of emotions have also led to some interesting research and most recently this has led to some mainstream products with spin off from eg MIT \cite{Affectiva}. In the aspect of serious games for autism Helmi Adly Mohd Noor, Faaizah Shahbodin and Naim Che Pee gives a good review of related literature \cite{Review}. The most familiar impacts of ASD (autism spectrum disorder) is in social cognition, communication and imagination. Autism is a very well researched area within science and can be divided into five categories: autistic disorder, asperger's syndrome, rett's disorder, childhood disincentive disorder and pervasive development disorder\cite{Review}. 
The frequency of the disorder in Norway is one in a thousand and growing \cite{Stat}. One important aspect for people with ASD is their difficulties to communicate in social life. Research indicates that individuals with autism show no clear deficits when matching basic expressions or detection subtle expressions when presented for a longer duration. However, they appear to have difficulties when they are processing simple expressions rapidly and discerning subtle differences in complex expressions \cite{weigelt2012face}.

\subsection{ADS case study}\label{ref:caseStudy}
The solution described in this article is categorized as an edutainment game (see section \ref{ref:classification})  with the ability to motivate and engage people with ASD and thereby potentially better their understanding of emotions \cite{zeeland2010reward}. By attaching game mechanics in the development of the game, it will provide both entertainment and education of the different emotional states, sensus. Social cognition is one of the important factors while dealing with autism. The purpose of the game design  is to enhance self-efficacy. Additionally, the solution enables the possibility to enlarge, extend, exchange the context of use. 
The contribution to science is to provide a solution that can be useful in various edutainment context, and in this case a face recognition solution to provide possible enhancement of social and communication adaptations for users with autism \cite{weigelt2012face}. 


\subsection {Effectiveness of Visual education in Autism}\label {ref:Cost}
The Fleming’s VARK model presents some commonly used learning styles \cite{fleming2001vark}. It appears that autistic individuals are more likely to rely on only one style of learning. It's beneficial to find out what kind of learning style the person favours. If the person likes to watch video, looking at books with pictures or observing persons, the visual learning style might be right. For those using their hands a lot puzzling with objects and moving things around, it may indicate a 'hands on' learner. The auditory learner tends to talk a lot and enjoy conversations and like to listen to music. Most people with ASD tend to be visual learners. Linda Hogden \cite{Quirk} regards through observations   that:
\\
\\
 "It is best to think of people with autism “as being 90\% visual and 10\% auditory learners".
\\
\\
 In Sintavilla there will mostly be a visual leaning style although some text will be attached explaining the storytelling using a auditory learning style for the following reasons:


\begin{itemize}
\renewcommand{\labelitemi}{$\bullet$}
\item Enjoy interacting with technology and video games\cite{Review}.
\item Visual skills of people with autism are often superior to their skills in other areas. 
\item People with autism claim to use visual information to interpret their world.
\item People with autism are better able to comprehend permanent (non-transient) visual information because the message is present long enough for them to take in and process the information.
\item Programs that use visual strategies are highly effective with people who have autism.


\end{itemize} 

\section{Relevant solutions}\label{ref:solutions}
In order to contribute to the development in serious games for health, focusing on autism, it's necessary to review other similar solutions. Teacch should be mentioned in this setting. Teacch is not a single technique or method but a complete program of services for autistic people which makes use of several techniques and methods in various combination depending upon the individual person's needs and emerging capabilities\cite{Teacch}. The proposed solution in this development should be seen as a potential tool within this process not as a complete solution on its own. Relevant solutions are cMotion\cite{finkelstein2009cmotion}, Let's Face It\cite{tanaka2010using}, SmileMaze\cite {cockburn2008smilemaze} and an interactive toys \cite{barakova2007using} approach. They all focus on autism and show an increasing use of new technology to interact, motivate and make the users more aware. Some show good result and others need a deeper study to display its benefit, although they all reveal a need for good an innovative solutions in the area.  

%\begin{figure}[ht]
%\centering
%\includegraphics[width=5in]{Diagram.png}
%\caption{{\small Relevant solutions}} \label{fig:compare}
%\end{figure} 



\begin{itemize}

\renewcommand{\labelitemi}{$\bullet$}

\item By giving the person means to understand better his/her environment, it (the environment) becomes more predictable and less anxiety generator. This may require proposing a simpler environment in the early phases of development and progressively reintroducing complexity as the child progresses towards more and more autonomy\cite{Teacch}.

\item By giving means of communication to the person the comprehension and expression capabilities will enable him/her to understand better what is being told/asked and to express his/her needs and feelings by other means than behaviour problems\cite{Teacch}.

\end{itemize} 

\subsection{cMotion}
Uses Virtual humans and programming logic to teach emotion recognition to children\cite{finkelstein2009cmotion}.
By introducing avatars, cMotion hopes to increase users gaming motivation and therby their ability to learn. 

\subsection{Interactive toys}
Emilia Barakova et.al \cite{barakova2007using} uses a interactive toys in order to motivate and make children with ASD more aware. The children also showed a exploitative behaviour in using the toys. They targeted 5 to 9 years old in a controlled environment disconnected from strangers, though ASD children tend to become inhibited when interacting with strangers. They test the children’s interest for pattern and regularity.  

\subsection {Let's face it}
The program is a multimedia, computer-based intervention that is designed to teach face processing skills to children with autism. 
The aim is to develop effective treatments that will enhance the face processing skills of children with autism spectrum disorders (ASD) as well as other populations that exhibit impairments in facial processing \cite{tanaka2010using}.

\subsection{SmileMaze}
The goal of SmileMaze is to improve the expression production skills of children with
ASD in a dynamic and engaging game format\cite {cockburn2008smilemaze}. CERT automatically detects
frontal faces in a standard web-cam video stream\cite{littlewort2006dynamics}. SmileMaze integrates the use of
facial expression production into an intervention program
aimed at improving the facial expression recognition and
production skills of children with ASD. SmileMaze is intended to become a part of the Let's Face It program. 





\section{Sintavillie}


The first step for this project was to make a face recognition system using pictures where the user gives a feedback on the actual emotion from the facial posture trying to recognize positive or negative valence\cite{krathwohl2002revision}. The term valence is used to categorize positive and negative emotions in the field of psychology. 

The second step includes android faceRecognition API and opencv \cite{OpenCV}. This give the capability for the user to take pictures using hardware in a mobile device from a given state of human emotion. Autism have many stages of illness and it would be to much of a challenge to implement a solution to cover them all. 

The game focuses on motivating patients with moderate autism with a expected potential result to better their social interaction. The game benefits have to be studied in order to state its characteristics of a serious game for health or a serious game for education \cite{weigelt2012face}.  



\subsection{Game mechanics}\label{ref:mechanics}
There are different game mechanics implemented in this game. The skinner box \cite{SkinnerBox} is used by awarding players with points when they have collected enough smiles. As the player meet a certain threshold an award will be given, representing another game mechanic: Achievements \cite{Achivements}.

\begin{figure}[H]
\center
\includegraphics[width=90mm]{begge.png}  
\caption{{\small Representation of level 1 and 2 in Sintavillie, respectively.}} \label{fig:Emotion}
\end{figure}


As the player reaches a threshold,  achievements are gained. These mechanics are implemented in order to make the player follow the pre-set path of the game and motivate into playing for a longer time. Wikipedia links throughout the game makes use of tangential learning \cite{TangentialLearning}. The target group is people with ASD and thereby the wikipedia input is asserted to their educational needs. 


\subsection{Game design}
As mentioned, a regular pattern in game play is advisable when motivating people with ASD. The storytelling, level progress and game mechanics will be affected “to keep it simple” for the user.

Humans have a huge number of emotional expressions, although in order to follow the stated development rules, only smiling and not smiling will be recorded for the game play. An introduction to the different levels eg. where the user takes pictures of their family, with a short description of the different types of emotional faces to 'catch' beforehand. The introduction will provided some picture samples that will illustrate different emotions. There will be a focus on two types of emotions; happy and angry. The user will be provided with extra information on how to play before the initiation of the camera (eg. how to place faces inside the rectangle). When the tutorial is completed, users will be able to start the game.
Based on the adventure genre representing a sustain structure\cite{Simen} to align with the learning goal:
\begin{itemize}
\renewcommand{\labelitemi}{$\bullet$}
\item Background story - the game universe
\item Task description - what you need to solve
\item Mind game, puzzle
\item Assessment - a type of test
\item Debriefing - a summery of achievements

\end{itemize} 


\subsection{Serious game classification}\label{ref:classification}
In this project an offline solution for an Android OS device with touch screen and integrated camera will be implemented. Even though the game solution is targeting health/therapy, in the way of improving ASD patients social skills and thereby how they communicate to others, there will not be made any statement that the game actually improves the users health. A edutainment classification \cite{Review}might be more appropriate until a further study on the games health improvements and how it correlate to social improvements for people with ASD. 
Furthermore the solution can be used in various context. Therefore people with autism will be used as a case-study to narrow the area of development. The underlying technology and research area can lead to different development directions. There are recognition of the whole body or part of a specific region of the body. This project will focus on emotional recognition from the human face. Besides the latter a good development strategy would be to look at regular patterns, heuristic evolution and user centred processes.       


\subsection{Scenario}\label{s:scenario}

The game will be built on history where a hero, user, will protect the island called “Sintavillie". The objective of the game is to collect 

as much as possible happy faces, here actually it will be focused on emotions. The “Hero” uses emotions and faces to make Sintavillie happier and peaceful. In some levels there will be need of collecting angry faces beside general happy faces see appendix \ref{ap:b}. The angry faces can be used to mask hero in the enemies castle to get near to the King Sint who try to spread the sadness and angry people in the Sintavillie. The pointing system of the game will use Skinner box idea, where user will gain points if he succeeds and lose points if wrong faces have been collected. At the end of game as at the start of each level and game there will be comics that will describe the process of game. The game will have a linear story narrative with the user in 1 person perspective. Different levels will help user with self-efficacy that will  affect human endeavour \cite{conner2005predicting}. The figure below uses the narrative way of presenting the story. 

\begin{figure}[H]
\center
\includegraphics[width=40mm]{test5.png}\phantom Q\includegraphics[width=40mm]{test4.png}
\caption{{\small Screen shot for narrative way of telling a story.}}
\end{figure}


\subsection{Benefits from new technology}
In a study from 2010, Kurt Squire \cite{squire2010information} shows in an educational game how new technology can be used to extend the classroom and prepare students for future participation in the real world. This is what people with ASD struggle with on a daily basis. Squire talks about an approach where new technology used by students can contributes to the research knowledge base. New technology can not only make this contribution more efficient but also give researchers instant access to data \cite{miller2012smartphone}\cite{chen2011service}. The data is essential in providing useful and effective feedback back to the users The huge amount of information online also gives a great posibility for up to date tangential learning (see section \ref{ref:mechanics}).

\section {Limitations}
The solution can be installed on off the shelf Android phones, SDK api level 8 and android version 2,2  or newer. There is no need for, or is the current version, suited for online usage at this stage of development. An online extension will increase the scientific contribution as well as the potential social interaction to other players. Although similar solutions (see section \ref{ref:solutions}) show good result in motivating and heightened awareness,  a study is needed to confirm the hypothesis for this solution. This will implicate a need to perform testing on people with ASD. 
\\
Another social aspect with the game, is the possibility to interact outside the classroom or home. By interacting with strangers, people with ASD might learn how to become less inhibited. 

\section {Cost}
There is always a cost involved in any development and serious games are usally cost more developing than games in general. The price will depend a lot on what technology, scope, variation and game genre \cite {Simen}.  A universal question is cost vs result. What will the overall cost be in developing the final result and will the benefits outweigh the increased health effect compared to more traditional methods. A draft of the economical costs are outlined in appendix \ref{ap:c}.


\section {Results}

Some people could gain knowledge from the Sintavillie game while others find it to be a game for entertainment. The goal is to converge the functionality to adapt to the category or serious games and thereby focusing on people with difficulties to interpret facial expressions. For this reason we focus on people with Autism, although it can be used for other similar cases.

Using the mobile camera to interact and read other peoples emotions, most highly increases the social skills for people with ASD, and the variety of contexts it can be used in, enhances the social experience compared to other only computer based solutions. 
\begin{figure}[H]
\centering
\includegraphics[width=60mm]{test1.JPG}\includegraphics[width=60mm]{test2.JPG}\\
\includegraphics[width=60mm]{Lars.JPG}
\caption{{\small Recognizing facial expressions. The squares on the picture can be hidden and the white dot can be used to verify that the facial expression is found (in this case a smile, se appendix \ref{ap:b}.}} \label{fig:Emotion}
\end{figure} 

The implementation functionality for the mobile device was done by using the OpenCV library and Android API. In order to get a better understanding, and retrieve emotional recognition results, the implementation was done for the human face. The different part of the face was separated in order for the application to detect the face, eyes and mouth. In the end the smile recognition functionality also got a successful implementation. The incremental development and implementation of the different functionalities, resulted in a more efficient code, that can be used for different purposes. Emotions have unalike features and there will also be a need to implement a possibility to detect other emotions eg. by detecting eyebrows together mouth in order to detect angriness. Because of this, the development approach used in this project can easily integrate new features (see section\ref{ref:caseStudy}). 
\\
The images in figure \ref{fig:Emotion} illustrate different tests done on random images and persons. They show the functionality available to recognize smiles from images and real people.


The images have rectangular lines that indicates different parts of the face. The detected parts, like the mouth, is indicated with a white dot to represents a successful smile recognition. If the dot is not there, the recognition is not successful. The lines are good for testing in the development stage and some users might even find them useful in order to play the game. They are optimal and can be  visually removed.

The beta version of the application is available in three levels. The full draft includes several appendixes: levels in the game, issues with the OpenCV technology and an estimate for development cost for the project.


\appendix{}
\section*{Appendicies}
This section includes all related files to the project and final report.



\section{Appendix}\label{ap:a}
\subsection *{OpenCV}
The official web page of OpenCV \cite{OpenCV} was the initial step  when setting up OpenCV in Eclipse IDE for the Android development. The documentation explained a step by step procedure for the integrating of the OpenCV library with the chosen IDE. Unfortunately, due to some changes in Mac environment, the NDK and make command for building the C/C++ code was failing. Therefore, the correct path with references had to be set manually in the .bash\_profile file. The second issue when setting up the environment was due to OpenCVs manager package flaw installing the emulator. This was solved by installing the appropriate manager package referenced in the tutorial. As the team furthered ahead it faced next difficulties. The deploying of apk was failing due to wrong reference in Android.mk file in JNI folder. Some other minor problems occurred due to different referencing of library.

\section{Appendix}\label{ap:b}
\subsection *{Plot}
Plot:
\\
Far in the north there is an island "Sintavillie" full of sadness. But long time ago it was a blooming island called Heldievillie before it was occupied by King Sint. Therefore, you must save people of Sintavillie collecting smile-faces and spread them in the island. Share smile, help people once again be able to smile and evolve happiness.

\subsubsection *{Levels}

Levels:
\\Level 1: Random of images with emotions: Map emotion, smile, from given images. if user finds the right image and choose it then he gets a text that tells that he did right otherwise there will be a text with a message that asks him to try again.
\\
\\Level 2: Collect smiling faces: Take picture of people around you, ask them to smile. If specified emotion is recognised user should take a picture. The OpenCV algorithm checks if the emotion is right one, if so it takes a picture and the user earns points, this will terminate camera and he will return to the level list with an ability move on further. The new level will appear. Otherwise he will still be at the same position and camera will indicate that he/she has to try again.
\\\\Level 3: Are you smiling? : Take picture of yourself to check if you are smiling. (same for being angry)
\\\\Level 4: My Family: Collect pictures that you will take to map individuals according to text. My daddy, my mom my brother…....granny... 
\\\\
Level 4a: My family smiling: Extra Images where everyone is smiling.
\\\\
Level 4b: My family angry: Everyone is angry.
\\\\Level 5: Recognize your family: Taken images in Level 3 and sub levels can be used for asking various questions like: Is it your daddy on image? Does your mom smile?.... etc
\\\\Level 6: Reversed or inverted?: Face inversion detection. 
\\\\Level 7: Avatar makes same emotion as me 
\\… .

\section{Appendix}\label{ap:c}
\subsection *{Cost}
Employees: 2 developers. Yearly cost 350.000.-
\\
Extra help: 2 professors. Cost per hour 1000.-
\\
\\
Including social expences:
\\
350.000.- x 1,32 = 462.000.-
\\
\\
Including material and working cost:
\\
462.000.- x 1,4 = 646.800.-
\\
\\
Total yearly cost developing = 1.293,600.-
\\
Estimate for 3 month developing = 323,400.-
\\
Ekstra help (consulting) = 12 hours x 2 professors x 1000.- = 24.000.-
\\
Total sum for project = 323,400 + 24,000.- = 347,400.- 



\section{Appendix}\label{ap:d}
Let's face it is based on an research based framework\cite{tanaka2003framework}.
Uses a domain model to describe face recognition and how people with ASD can train to get better. \\
Domain I: Attending to faces \\
DomainIIa: Facial identity\\
DomainIIb: Facial Expression\\
Domain III: Social meaning of facial cues\\
\\
Specific vategorization of perceptuell expertice \\
\\
New objects more easily recognized\\
New objects more easily learned\\
\\
Specific levels of categorization




%NOTE: Run bibtex build twice and after latex build to have correct output
%below the bibliography is used for the setting up citation
\bibliographystyle{IEEEtran}
%imports data from references.bib file
\bibliography{AutismDatabase}

\end{document}
